DGLab Question Answering System and Automatic Evaluation Method at NTCIR-13 QA Lab-3 for University Entrance Exam on World History Essay

The paper describes DGLab question answering system and automatic evaluation method at NTCIR-13 QA Lab-3 for Japanese university entrance exam on world history essay. Submissions of subtasks include extraction, summarization, and evaluation method, in Phase-2 and Research Run for both Japanese and English. The proposed system follows the organizer-recommended pipeline, consisting of an extraction module and a summarization one. The former comprises a condition extraction routine, a full-text search engine, and a sentence selection heuristics. The latter one utilizes wellestablished summarization algorithms that extract sentences based one their importance. For evaluation method, an independent system measures semantic similarity between submitted summaries and gold standard essays. While particularly on summarization subtask, the submissions have gained competitive performance among participants, in terms of total scores graded by human experts. The quality of the machine-generated essay, however, is far from ideal to pass the exam. A brief discussion is therefore presented to diagnose symptoms and their probable causes. Categories and Subject Descriptions • H.3.4 [INFORMATION STORAGE AND RETRIEVAL]: Systems and Software Performance evaluation (efficiency and effectiveness), Question-answering (fact retrieval) systems.